Generalizability, Replicability, and New Insights Derived From Registered Reports Within Understudied Populations

Free AccessGeneralizability, Replicability, and New Insights Derived From Registered Reports Within Understudied PopulationsIntroduction to the Special IssuePeter A. Edelsbrunner, Kai Ruggeri, Kaja Damnjanović, Samuel Greiff, Jérémy E. Lemoine, Matthias ZieglerPeter EdelsbrunnerPeter ETH Zurich, RZ H16, Claussiusstrasse 59, 8092 [email protected] Department of Humanities, Political Social Sciences, Switzerland Search for more papers by this author, Ruggeri Mailman School Public Health, Columbia University, York City, NY, USA Damnjanović Faculty Philosophy, Laboratory Experimental Psychology, Institute University Belgrade, Serbia Greiff Behavioral Cognitive Luxembourg, Luxembourg Lemoine East London, UK ESCP Business School, Ziegler Humboldt-Universät zu Berlin, Germany authorPublished Online:December 08, 2022https://doi.org/10.1027/1015-5759/a000743PDF ToolsAdd favoritesDownload CitationsTrack Citations ShareShare onFacebookTwitterLinkedInReddit SectionsMoreResearch on assessment that takes an individual differences-perspective is definition dependent individuals specific populations it studies. Consequently, if we study from same or similar underlying over again, will miss important insights stability findings, how psychological traits are structured play out in with different characteristics across populations. This not only limits results population investigated but also involves a high possibility type 1 errors overgeneralizing findings their implications beyond empirical rationale.There initiatives regions globe aim deal challenges posed lack generalizability replicability (Bishop, 2019; Earp & Trafimow, 2015). These include international national networks researchers aiming promote research (e.g., Center Open Science, Reproducibility Network), encouragement pre-register studies (Eich, 2013; Van’t Veer Giner-Sorolla, 2016), replicate previous Visser et al., 2022), free online platforms share protocols data Science Framework). In line these initiatives, motivation behind special issue EJPA (Ruggeri 2019) encompass contributions sometimes neglected populations, combine aspect approach registered reports (Greiff Allen, 2018). way, intended gather tackle issues generalizability, replicability, within designs.As has been pointed repeatedly perspectives, two interwoven issues, applies field assessment. Fiedler (2011) described samples all other aspects might induce variation commonly picked ways maximize observed effects. Whereas allows better pin down theoretically expected effects, can undermine Barr colleagues (2013) context statistical modeling. They argued order should specify random models factors variability design.Their arguments, however, do have implications; they point toward theoretical consideration often-neglected design sampling. Specifically, want generalize meaningful our studies, then effects represent means establishing generalizability. Similarly, Yarkoni (2020) specification variables could variation. Approaches which many analysts model shown researcher degrees freedom analysis contribute conclusions very Hoogeveen 2022; Schweinsberg 2021).In present issue, aimed would due assessed as well researchers’ conducting analyzing data. was achieved assessing focus respective topics prioritizing were pre-registered reports. Trialing, extending, comparing between seldom help us question long-standing beliefs regarding untested assumptions Terracciano 2005). Preregistration methodological quality turn improve reliability (Soderberg 2021). With foci, introduces set various readership journal. The hope readers (future) authors see valuable tool increasingly finds its way into toolbox assessment.How Present Papers Contribute Shed Light Generalizability RobustnessIn addition specifically selected doing so mostly basis reports, found tackled questions relevance community. section, summarize what five did how, top question, understanding replicability.Twomey Johnson (2022) created norm tables Ireland International Personality Item Pool (IPIP; Goldberg 2006) NEO-300. Irish participants (N = 18,591) global IPIP-NEO-300 dataset Johnson’s IPIP-NEO repository identified norms personality facets age groups. investigating using interpret own participants. Although article based report (unlike four articles published issue), paper included make convincing case providing United Kingdom open source instrument (i.e., IPIP-NEO-300). resources who financial constraints cannot (or rather not) use proprietary measures Big Five. study, inclusion freely accessible modifiable tables, contributes science integrates needs little resources. Therefore, believe fully scope aims issue.Rachev examine whether risky-choice attribute framing effect, describing preference under presentations problem (Tversky Kahneman, 1981), occurs shows relations willingness ability think rational samples. represented constructs; namely, actively open-minded thinking (Haran 2013) bullshit receptivity Ilić susceptibility associated constructs both samples, stronger negative relation Bulgaria than North America. Rachev al.’s (this issue) several ways. First, each construct measured multiple items tasks, allowed indicators at broader level when latent variable Second, performed measurement invariance analyses; enabled them identify correct cross-cultural deviation perform comparison invariant model. Finally, exemplifies incorporate preregistered procedures evaluating fit (cf. 2018), central statistics implemented.Clay investigate relationship willpower perceived effort, dispositional (personality traits), satisfaction reward, topic may be interesting cognitive psychology, decision sciences, differences, neurosciences general. main hypothesis people holding limited perceive tasks effortful rewards less satisfying relative hold such beliefs. 187 America, multilevel indicate support hypothesis. addition, find higher levels need cognition presented (N-back tasks) effortful, whereas there small comparably few Five domain. On report, demonstrates preregistration ensures involving mental depletion allegations p-hacking recent years (Friese 2019). does involve experimental manipulation depletion, preregistration, supposed limit questionable practices, self-ratings related produce clear patterns correlations.Petrovic Žeželj development initial validation scale proneness doublethink, define general tolerate inconsistencies one’s knowledge. new approximates one-factor structure among consistent conspiracist mentality, conspiracy theories, intuitive styles, Similar (2022), evaluation robust phenomena measure belief Serbian context. relationships concept doublethink theories thinking. result points including culture-specific tailored non-WEIRD samples.In last contribution, Buabang validate measuring well-being six European countries US. scale, contrast hypotheses, unidimensional showed two-dimensional affective behavioral stress. further given partial invariances achieved. country comparisons undertaken deviations, conclude composite scores used valid comparisons. contribution reporting appear suboptimal perspective (meaning analytical approach) a-priori-defined analyses, adding trustworthiness. Twomey highlights importance (Vandenberg Lance, 2000) greater generalization findings. vital policy decade Scherer, 2018; Bensch, exemplified current findings.What Do We Learn Contributions?In acting guest editors Journal Psychological Assessment, learned great about projects assessment, role therein. As highlight regard implicitly assumed processes capturing work independently socio-cultural contexts. definitely case. Studies examining actual provide top-down tests questioning generality; bottom-up-development boundary conditions reliable phenomena.Finally, perhaps especially noteworthy, most conducted teams diverse countries. Many contributing still pre-or post-graduate phases during active research. makes even challenging engage long-term commitment usually come along with. At time, project provides optimal academic playground connections junior look established paradigms local Assessment experts institutions follow strongly diverging inconsistent approaches. An arguably prime example views item response theory Rasch modeling take Edelsbrunner Dablander, Junior academics educated history grounded Response Theory learn IRT, those focusing distinct foundation methods. showcases that, although partially formal theory, highly incongruent perspectives inform one another progress. By gathering presenting inspire consider broadening networks, affiliate academics, typical borders state, age, identity, engaging collaborative activities foster transparency replicability.References Barr, D. J., Levy, R., Scheepers, C., Tily, H. J. (2013). Random confirmatory testing: Keep maximal. Memory Language, 68(3), 255–278. https://doi.org/10.1016/j.jml.2012.11.001 First citation articleCrossref, Google Scholar Bishop, (2019). Rein horsemen irreproducibility. Nature, 568(7753), 435. https://doi.org/10.1038/d41586-019-01307-2 Buabang, E., Ashcroft-Jones, S., Esteban Serna, Kastelic, K., Kveder, Lambertus, A., Müller, T. K. (2022). Validation Personal Financial Wellness Scale: A multinational 7 Assesment, 38(6), 476–486. https://doi.org/10.1027/1015-5759/a000750 articleAbstract, Clay, G., Dumitrescu, Habenicht, Kmiecik, I., Musetti, M., Domachowska, I. Who satisfied effort? Inter-individual differences determinants perception effort reward. 452–462. https://doi.org/10.1027/1015-5759/a000742 Earp, B. D., (2015). Replication, falsification, crisis confidence social psychology. Frontiers 6, Article 621. https://doi.org/10.3389/fpsyg.2015.00621 P. F. psychometric scientific reasoning: review recommendations future avenues. Educational Psychology Review, 31(1), 1–34. https://doi.org/10.1007/s10648-018-9455-5 Eich, usual. 25(1), 3–6. https://doi.org/10.1177/0956797613512465 Fiedler, (2011). Voodoo correlations everywhere – Not neuroscience. Perspectives 6(2), 163–171. https://doi.org/10.1177/1745691611400237 Friese, Loschelder, Gieseler, Frankenbach, Inzlicht, M. Is ego real? arguments. 23(2), 107–131. https://doi.org/10.1177/1088868318762183 Goldberg, L. Johnson, Eber, W., Hogan, Ashton, Cloninger, C. Gough, G. (2006). pool public-domain measures. Research Personality, 40(1), 84–96. https://doi.org/10.1016/j.jrp.2005.08.007 S. (2018). submission format. 34, 217–219. https://doi.org/10.1027/1015-5759/a000492 articleLink, R. Still apples oranges? 34(3), 141–144. https://doi.org/10.1027/1015-5759/a000487 Haran, U., Ritov, Mellers, information acquisition, accuracy, calibration. Judgment Decision Making, 8(3), 188–201. https://doi.org/10.1017/CBO9781107415324.004 articleGoogle Ilić, (2021). effect credibility receptivity. Applied 35(5), 1193–1205. https://doi.org/10.1002/acp.3852 Hoogeveen, Sarafoglou, van Elk, Wagenmakers, E.-J. Many-analysts religiosity [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/pbfye Petrovic, B., Žeželj, Thinking inconsistently: Development doublethink. 463–475. https://doi.org/10.1027/1015-5759/a000645 Rachev, N. Geiger, Vintr, Kirilova, Nabutovsky, Nelsson, Actively thinking, receptivity, framing: dual-process account America Bulgaria. 440–451. https://doi.org/10.1027/1015-5759/a000685 Ziegler, Call “Advancing reproducibility populations”. 35(2), 295–296. https://doi.org/10.1027/1015-5759/a000523 Schweinsberg, Feldman, Staub, N., den Akker, O. Aert, Assen, Liu, Y., Althoff, T., Heer, Kale, Mohamed, Z., Amireh, H., Venkatesh Prasad, V., Bernstein, Robinson, Snellman, Amy Sommer, Otner, … Luis Uhlmann, Same data, conclusions: Radical dispersion independent operationalize test Organizational Behavior Human Processes, 165, 228–249. https://doi.org/10.1016/j.obhdp.2021.02.003 Soderberg, Errington, Schiavone, Bottesini, Thorn, Vazire, Esterling, Nosek, Initial evidence compared standard publishing Nature Behaviour, 5(8), 990–997. https://doi.org/10.1038/s41562-021-01142-4 Terracciano, Abdel-Khalek, Adám, Adamovová, L., Ahn, Alansari, Alcalay, Allik, Angleitner, Avia, Ayearst, Barbaranelli, Beer, Borg-Cunen, Bratko, Brunner-Sciarra, Budzinski, Camart, Dahourou, McCrae, (2005). National character reflect mean trait 49 cultures. 310(5745), 96–100. https://doi.org/10.1126/science.1117199 Tversky, (1981). decisions psychology choice. 211(4481), 453–458. https://doi.org/10.1126/science.7455683 Twomey, Open-source Ireland. 432–439. https://doi.org/10.1027/1015-5759/a000644 Veer, (2016). Pre-registration discussion suggested template. 67, 2–12. https://doi.org/10.1016/j.jesp.2016.03.004 Vandenberg, (2000). synthesis literature: Suggestions, organizational Methods, 3(1), 4–70. https://doi.org/10.1177/109442810031002 Visser, Bergmann, Byers-Heinlein, Dal Ben, Duch, Forbes, Franchin, Frank, Geraci, Hamlin, Kaldy, Kulke, Laverty, Lew-Williams, Mateu, Mayor, Moreau, Nomikou, Schuwerk, Simpson, Zettersten, Improving infant research: ManyBabies Brain 45, e35. https://doi.org/10.1017/S0140525X21000455 Yarkoni, (2020). crisis. e1. https://doi.org/10.1017/S0140525X20001685 Lost translation: Thoughts translation existing languages. 29(2), 81–83. https://doi.org/10.1027/1015-5759/a000167 ScholarFiguresReferencesRelatedDetails Issue: Advancing Across Borders PopulationsVolume 38Issue 6November 2022ISSN: 1015-5759eISSN: 2151-2426 Published onlineDecember 8, 2022 InformationEuropean 38, pp. 427-431 https://doi.org/10.1027/1015-5759/a000743.© 2022Hogrefe PublishingAuthorship:Peter shared head editorial shared, first authors.PDF download